7.2. Monitoring

Which dashboard is shipped?

Open http://localhost:3002/d/agentops-overview and watch the Agent request rate panel while you send a turn.

Check both host services before interpreting an empty dashboard:

curl -fsS http://localhost:9090/-/ready
curl -fsS http://localhost:3002/api/health | jq

The host Compose profile provisions AgentOps overview at that address with six metric panels and one logs panel:

sum(rate(agentops_calls_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(agentops_duration_seconds_bucket[5m])))
sum(rate(agentops_calls_total{status_code="STATUS_CODE_ERROR"}[5m]))
  / clamp_min(sum(rate(agentops_calls_total[5m])), 1)
sum(rate(agentgateway_requests_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(agentgateway_request_duration_seconds_bucket[5m])))
sum(rate(agentgateway_guardrail_checks_total{action="Reject"}[5m]))

These are request rate, p95 latency, error ratio, gateway rate/latency, and guardrail rejection rate. The Agent logs panel below them queries Loki for agentops-agent log lines, optionally filtered by the trace id textbox variable.

How are agent metrics produced?

Those panels are not a second instrumentation. The collector derives RED metrics — rate, errors, duration — from the spans you already exported in 7.1. Tracing.

The OTel span_metrics connector turns spans into request count and duration histograms under namespace agentops. It retains only bounded dimensions:

connectors:
  span_metrics:
    namespace: agentops
    dimensions:
      - name: gen_ai.operation.name
      - name: gen_ai.request.model
      - name: error.type

The scraper and UI differ by deployment profile. Use the canonical host/local/GKE matrix in 7. Observability before choosing a dashboard or external scraper; this page follows the host Compose path.

Where do my agent logs go?

To Loki, through the same collector that produced the metrics above.

After ADK configures its providers, the application installs exactly one OTel LoggingHandler on the agent logger when OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is configured. A trace-only endpoint, no endpoint, or OTEL_SDK_DISABLED=true installs none. Both collector profiles forward the resulting logs pipeline to Grafana Loki, which ingests native OTLP at /otlp/v1/logs:

logs:
  receivers: [otlp]
  processors: [memory_limiter, batch]
  exporters: [otlp_http/loki]

Loki runs pinned in single-binary mode with filesystem storage and 7-day retention, matching Prometheus: as a Compose service on the host (infra/observability/loki.yaml) and as a PVC-backed Deployment in Kubernetes (infra/k8s/base/loki.yaml). Grafana provisions the Loki datasource next to Prometheus; the Kubernetes overlay exposes only the loki:3100 ClusterIP, mirroring the external-scraper stance for metrics.

Content capture stays off by default, so ADK does not intentionally duplicate prompts or model responses into log records. Before OTLP export, the handler filters a copy of each record:

1. It locally redacts concrete PII and obvious credential/token patterns.
2. It caps every exported string and body at 2048 characters.
3. It removes exception messages, tracebacks, and stack bodies, and retains only exception.type.

Console handlers still receive the untouched local record. This is defense in depth, not a reason to log secrets; apply the same access and retention policy to Loki as to the trace store.

How do I jump from a trace to its logs?

Copy a trace id out of MLflow, paste it into Grafana, and you get that turn's log lines.

Log records emitted inside a span keep their trace context, and Loki stores trace_id as structured metadata — per-line fields kept next to the log text — on every OTLP-ingested line. That single id is the thread through all three pillars: one turn's OTLP export fans out at the collector, and trace_id reconnects the pieces at query time.

flowchart LR
    Turn["one agent turn<br/>(OTLP export)"] --> Col["otel-collector<br/>:4317 / :4318"]
    Col -->|traces| MLflow["MLflow trace<br/>:5000"]
    Col -->|"span_metrics<br/>connector"| Prom["Prometheus<br/>scrapes :8889"]
    Col -->|logs| Loki["Loki<br/>:3100"]
    Prom --> Graf["Grafana :3002"]
    Loki --> Graf
    Prom -->|alert rules| Alert["Alertmanager<br/>:9093"]
    MLflow -. "same trace_id" .-> Graf
    Loki -. "same trace_id" .-> Graf

To correlate one agent turn:

1. Send one request, open its trace in MLflow at http://localhost:5000, and copy the trace id.
2. Open http://localhost:3002/d/agentops-overview, paste the id into the trace id variable, and read the Agent logs panel.
3. Alternatively, query Loki from Grafana Explore or its HTTP API:

{service_name="agentops-agent"} | trace_id="<trace id from MLflow>"

curl -fsS -G 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="agentops-agent"}' | jq '.data.result | length'

In Kubernetes, forward the ClusterIP first with kubectl -n agentops port-forward svc/loki 3100:3100. The overlay ships no Grafana; query the forwarded API directly or point an externally operated Grafana at it.

Why avoid session or prompt labels?

Never label a metric with a user id, a session id, or a prompt.

Prometheus labels form an in-memory/index cardinality dimension: cardinality is the number of distinct label-value combinations the store must index. User ids, session ids, incident ids, prompts, or trace ids can make the store expensive and leak sensitive information. Keep correlation ids in traces/logs and metrics dimensions bounded to known model/operation/error values.

When should the platform page a human?

Page only on sustained, user-visible symptoms; everything else is a ticket-severity signal reviewed during working hours.

The shipped rules encode that split:

• page for error-budget burn against a 99% span-success SLO and for a dark telemetry pipeline.
• ticket for latency, missing token counters, guardrail spikes, and schema failures.

An error budget is the share of requests an SLO allows to fail — here, 1% of spans. The SLO alert requires both the 5m and 1h error ratios to exceed 14.4 times the 1% budget, plus at least three failures in 5m. One flaky request cannot page; three sustained failures can:

- alert: AgentErrorBudgetBurn
  expr: |-
    agentops:calls:error_ratio_rate5m > (14.4 * 0.010)
      and agentops:calls:error_ratio_rate1h > (14.4 * 0.010)
      and sum(increase(agentops_calls_total{status_code="STATUS_CODE_ERROR"}[5m])) >= 3
  for: 2m

On sparse lab traffic, drive at least three failed spans to exercise the page. Production should tune the absolute failure guard and burn windows from observed traffic rather than copying the lab threshold blindly.

What alerts ship with the course?

infra/observability/prometheus-rules.yml defines two recording rules (agentops:calls:error_ratio_rate5m/rate1h) and six alerts, each grounded in a metric the stack verifiably exports.

A recording rule precomputes a query and stores its result as a new metric series, so alerts can read one cheap value instead of recomputing a ratio.

Each row below carries the metric behind the rule, the for: window it waits before firing, and the response section to open when it does:

One spelling quirk in that Metric column: the collector's Prometheus exporter appends the token unit suffix, which is why the token counter reads agentops_tokens_token_total.

The host Compose stack loads the rules into Prometheus (http://localhost:9090/alerts) and routes fired alerts to a pinned Alertmanager at http://localhost:9093. That Alertmanager's webhook points at a documented placeholder (infra/observability/alertmanager.yml); replace the URL to integrate a real channel.

The local Kubernetes overlay runs the identical rules in an in-cluster Prometheus/Alertmanager pair (infra/k8s/overlays/local) whose Alertmanager receiver has deliberately no integration: default-deny egress keeps notifications cluster-internal, and network policies allow only Prometheus to reach it. There is no external paging service anywhere.

How do I respond when AgentErrorBudgetBurn fires?

1. Symptom: agent turns fail or return errors; the dashboard error-ratio panel rises with the alert.
2. Diagnose: inspect the failed turn and sanitized gateway logs for provider authentication, quota, or upstream errors. On the optional local profile, also check ollama ps; in Kubernetes, inspect kubectl -n agentops get pods.
3. Likely cause: invalid Gemini credentials, exhausted provider quota, an unavailable upstream, or a route/policy rejection. On Ollama, check model availability and the documented bind address.
4. Fix: repair the diagnosed provider configuration or gateway change; avoid repeated hosted retries while quota is exhausted. Confirm successful requests resume and the alert resolves.

How do I respond when AgentTurnLatencyP95High fires?

1. Symptom: turns complete but slowly; p95 stays above 15s.
2. Diagnose: open the slowest recent trace in MLflow at http://localhost:5000 and read which span dominates; separate provider latency from tool time and host pressure. Use ollama ps only on the optional local profile.
3. Likely cause: model cold starts/reloads, CPU contention with other workloads, or oversized contexts from long sessions.
4. Fix: keep the model warm, free host resources or pick a smaller pinned model, and trim session growth; if your hardware is simply slower, raise the threshold in the rules file instead of deleting the alert.

How do I respond when ObservabilityCollectorDown or AgentTokenTelemetryMissing fires?

1. Symptom: dashboards flatten while the agent still answers — the pipeline, not the agent, is broken.
2. Diagnose: docker compose -f infra/observability/compose.yaml ps otel-collector (host) or kubectl -n agentops get pods -l app.kubernetes.io/name=otel-collector; then curl -fsS http://localhost:8889/metrics | grep agentops_tokens after a port-forward to see whether the counter is exported at all.
3. Likely cause: collector crash or memory-limit kill, a port conflict with the other profile (do not run host Compose and the forwarded in-cluster stack together), or an agent started without OTEL_EXPORTER_OTLP_ENDPOINT so spans arrive from one process and metrics from none.
4. Fix: restart the collector, resolve the port clash, and relaunch the agent with the documented OTLP environment; telemetry gaps for the outage window are permanent, which is exactly why this pages.

How do I respond when a guardrail or triage-schema alert fires?

1. Symptom: AgentInjectionNeutralizedSpike or AgentTriageSchemaFailures shows up as a ticket; user traffic may look normal.
2. Diagnose: filter Loki for the recent turns ({service_name="agentops-agent"}), open the matching MLflow traces, and inspect which tool output carried injection markers or which report failed validation.
3. Likely cause: for injections, adversarial content in the data the tools read (or someone running the red-team suite); for schema failures, model or prompt drift after a model swap.
4. Fix: for injections, confirm the neutralization worked and clean or quarantine the offending source records; for schema failures, re-run the offline tests and evaluation gates and restore the pinned model/prompt combination before trusting new reports.

How do you query the stores directly?

When a panel disagrees with what you expect, read the raw endpoint instead of the dashboard. These are lookup commands, not first-pass reading: come back to them when a graph and a trace tell you different stories.

curl -fsS 'http://localhost:9090/api/v1/query?query=sum(rate(agentops_calls_total%5B5m%5D))' \
  | jq '.data.result'
curl -fsS http://localhost:15020/metrics | head

kubectl -n agentops port-forward svc/prometheus 9090:9090

kubectl -n agentops port-forward svc/alertmanager 9093:9093

kubectl -n agentops port-forward svc/otel-collector 8889:8889

curl -fsS http://localhost:8889/metrics | head

How much load can the platform take?

This section and the next two are the page's second half: a longer, separate exercise that measures the same path under load.

k6 is a load-testing tool. It drives scripted HTTP traffic and fails the run when a stated latency budget is breached. It counts concurrency in VUs — virtual users, one simulated client each.

The repository ships Grafana k6 scenarios under load/ — k6 is AGPL-3.0 open source, consistent with the rest of the stack. Each script isolates one layer of the host quickstart:

1. load/health.js: raw /healthz on MCP :8000 and A2A :8080, plus a low-rate hop through agentgateway :3001 — the latency floor and the pure proxy overhead.
2. load/mcp-read.js: the MCP streamable HTTP handshake and a tools/call list_incidents loop through the gateway :3000 — gateway plus the MCP server plus SQLite, with no model call.
3. load/a2a-send.js: a bounded A2A message/send conversation through :3001 — a full agent turn, model included, deliberately capped at 1 VU and 3 iterations. It requires a completed, non-empty result with no structured ADK error; HTTP 200 alone is not success.
4. load/fake_model.py: a deterministic OpenAI-compatible response on Ollama's host port, so the same A2A turn measures the platform without inference.

First run the isolated mise run smoke:host composition check.

Follow 5.1. Gateway Setup for the manual host stack and A2A gateway environment. The default model route uses hosted Gemini. Health and MCP tests call no model; run the A2A sample only when you intend to use provider quota:

mise run load:health
mise run load:mcp
ITERATIONS=1 mise run load:a2a # one real agent turn; model calls may be billed

For a comparison without inference, stop the A2A process and gateway and free :11434 if your Ollama process owns it. Start mise run model:fake and mise run gateway:host:ollama in separate terminals. Then restart A2A from agents/python/:

AGENT_MODEL_PROVIDER=openai-compatible \
AGENT_MODEL=qwen3:4b-instruct \
OPENAI_BASE_URL=http://127.0.0.1:4000/v1 \
OPENAI_API_KEY=local-gateway \
AGENT_MCP_URL=http://127.0.0.1:3000/mcp \
AGENT_A2A_STREAMING=false \
mise run a2a

Run the same load:a2a task from the root. The explicit Ollama gateway profile sends requests to the fake on :11434; leaving gateway:host active would still use hosted Gemini.

The fake returns fixed text and token usage without tool calls. It measures a simpler platform path, not an identical real-model trajectory or answer quality. Stop the fake and restart your chosen gateway/provider configuration before evaluating behavior.

The A2A checks validate protocol state as well as transport status. A direct Message must contain text. A Task must reach completed, contain text in its status message or artifacts, and carry no metadata.adk_error_code. A failed task wrapped in a successful JSON-RPC response therefore fails the sample instead of producing a false latency result.

The first honest answer is that the shipped gateway policies cap throughput on purpose: 120 MCP, 60 A2A, and 30 model requests per minute per gateway instance. The default rates stay under those budgets, and mcp-read.js counts every HTTP 429 in an mcp_rate_limited metric whose threshold is zero. A breach means you measured your own rate limiter, not the platform.

For a real capacity probe, raise maxTokens in the gateway config, or point MCP_URL at the raw :8000/mcp server to take the gateway out of the path. In the local Kubernetes overlay, run the identical scripts against the port-forwarded agentgateway and raw services and watch pod CPU limits in parallel.

What is a latency budget?

A latency budget is a pass/fail number agreed on before the test: this path answers within X ms at percentile Y.

Deciding afterwards that a dashboard looked fine is not a budget. k6 encodes budgets as thresholds, so a breach fails the run with a non-zero exit code, exactly like a failing unit test. From load/mcp-read.js:

thresholds: {
  // Latency budget — a starting point for localhost, tune to your hardware.
  checks: ['rate==1'],
  http_req_failed: ['rate<0.01'],
  'http_req_duration{op:tools_call}': ['p(95)<250'],
  mcp_rate_limited: ['count==0'], // any 429 means the gateway budget, not the platform, was measured
},

The shipped starting points are:

1. p95 under 50 ms for raw health.
2. 100 ms for the gateway hop.
3. 250 ms for the MCP read.
4. 15 s for a full A2A turn.

The last number matches the alert threshold, but their measurements differ: k6 times complete HTTP turns, while the alert aggregates exported spans over a window. They can disagree. Read the percentiles from the end-of-run summary (http_req_duration ... p(95)=...): averages hide tail latency, and the tail is what a user waiting on an agent turn actually feels. On slower hardware, tune a breached budget deliberately instead of deleting it — the same instruction the alert rule carries.

Where does the time actually go?

Use the load samples to identify a slow path, then use correlated spans to locate time inside that path.

Subtracting independently measured p95 values does not yield the p95 of proxy or inference overhead. The samples can contain different requests, tool trajectories, connection setup, and cold starts. Three A2A turns are a smoke sample; report their observed latencies instead of treating the displayed p95 as a stable tail estimate.

To correlate a budget breach, start mise run observability:up before the run. The dashboard's gateway panels (agentgateway_request_duration_seconds) show the hop the gateway sees; the agent panels (agentops_duration_seconds) show time inside the process. A flat-fast gateway with a slow agent places the problem behind the proxy.

Then open the slowest turn in MLflow at http://localhost:5000 and read which span dominates. Filter Loki with {service_name="agentops-agent"} plus that trace id for errors or rate-limit rejections. It is the same three-pillar walk as any alert response above.

Your turn: how do you add an alert rule and its runbook?

Required drill — the Chapter 7 checkpoint asks for its result. Close the loop from a firing alert to a documented response.

• Mode: keep.
• Goal: add one new Prometheus alert rule tied to an observable outcome (e.g. sustained tool-error rate or p95 latency breach), and write the runbook an on-call engineer would open when it fires.
• Files to touch: infra/observability/prometheus-rules.yml, a new infra/observability/tests/<alert-name>.yml rule test, and a new infra/observability/runbooks/<alert-name>.md operator runbook.
• Preflight: choose a new alert/runbook name, require both new paths to be absent with test ! -e, and require git diff --quiet -- infra/observability/prometheus-rules.yml.
• Gate that proves completion: promtool check rules infra/observability/prometheus-rules.yml and promtool test rules infra/observability/tests/<alert-name>.yml both pass. The rule test supplies a fixed input series and proves the alert is inactive, pending for its declared window, firing with the exact runbook annotation, then resolved.
• Final state: keep only the rule, deterministic rule test, and operator runbook; git status --short shows no generated Prometheus data. Driving the condition on the host stack is optional runtime evidence and must be stopped when the observation is complete.

Keep this operator response outside agents/data/runbooks/. That directory is the fictional service's immutable retrieval corpus; adding the agent platform's own on-call instructions there would silently cross a trust and domain boundary.

What proves this page worked?

Work through these against the host Compose stack, in order:

1. Generate allowed, rejected, and failed requests.
2. Verify all six metric panels receive data, and compare the Prometheus result with the trace count.
3. Confirm labels contain no raw prompts, users, sessions, or trace ids.
4. Correlate one turn across all three pillars: its MLflow trace, the request-rate increase in Prometheus, and its Loki log lines filtered by that trace id.
5. Fire one alert deliberately: stop Ollama, send a few requests, and watch AgentErrorBudgetBurn move from pending to firing at http://localhost:9090/alerts and appear in Alertmanager at http://localhost:9093; restart Ollama and watch it resolve.
6. On the other profiles: in the local Kubernetes overlay, repeat against the forwarded svc/prometheus and svc/alertmanager; on GKE, stop at verifying the bounded metrics on :8889 and the forwarded Loki API unless you already operate a scraper.
7. Tear down with docker compose -f infra/observability/compose.yaml down, which preserves the named volumes; adding -v deletes the stored metrics and logs.

You are done when:

• All six metric panels on http://localhost:3002/d/agentops-overview show data after you send traffic.
• One trace id copied from MLflow returns that turn's lines in the Agent logs panel.
• No label on any panel carries a raw prompt, user, session, or trace id.
• AgentErrorBudgetBurn reached firing at http://localhost:9090/alerts, appeared in Alertmanager at http://localhost:9093, and resolved after you restarted Ollama.
• You have run at least one k6 script and read its p(95) line, so you know whether the shipped budget held on your hardware.
• Your own rule from the required drill passed promtool check rules, moved to firing when you drove its condition, and resolved when you cleared it.

Continue to 7.3. Costs when one trace id gets you from a metric spike to the log lines that explain it.